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Abstract 

This article introduces an important concept: Transparency by way of Humanistic Intelligence as a human right, and in 
particular, Big/Little Data and Sur/Sous Veillance, where “Little Data” is to sousveillance (undersight) as “Big Data” is to 
surveillance (oversight). Veillance (Sur- and Sous-veillance) is a core concept not just in human-human interaction (e.g. 
people watching other people) but also in terms of Human-Computer Interaction. In this sense, veillance is the core of 
Human-in-the-loop Intelligence (Humanistic Intelligence rather than Artificial Intelligence), leading us to the concept of 
“Sousveillant Systems” which are forms of Human-Computer Interaction in which internal computational states are 
made visible to end users, allowing users (but not requiring them) to “jump” into the computational feedback loop 
whenever or wherever they want. An important special case of Sousveillant Systems is that of scientific exploration: not 
only is (big/little) data considered, but also due consideration must be given to how data is captured, understood, 
explored, and discovered, and in particular, to the use of scientific instruments to collect data and to make important 
new discoveries, and learn about the world. Science is a domain where bottom-up transparency is of the utmost 
importance, and scientists have the right and responsibility to be able to understand the instruments that they use to 
make their discoveries. Such instruments must be sousveillant systems ! 
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Introduction 

Surveillance 1 (oversight, i.e. being watched) and sous¬ 
veillance 2 (undersight, i.e. doing the watching) can both 
be thought of in the context of control theory and feed¬ 
back loops. 3 

In particular, McStay considers surveillance in this 
way, i.e. in regard to the form of privacy that is inher¬ 
ently violated by profiling, and related closed-loop feed¬ 
back systems that manipulate us while monitoring us 
(McStay, 2011). Ruppert considers the interplay 
between surveillance and public space, through a case 
study of Toronto’s Dundas Square (Ruppert, 2006), 
where security guards prohibit the use of cameras 
while keeping the space under heavy camera surveil¬ 
lance. This surveillance without sousveillance 
(Mann, 2002a) creates a lack of integrity, i.e. surveil¬ 
lance is a half-truth without sousveillance (Mann et al., 
2015). 


The intersection of Sousveillance and Media was 
pioneered by Bakir, i.e. sousveillance as not merely a 
capture or memory right, but also sousveillance as a 
disseminational (free-speech) right. This gave rise to 
two important concepts: sousveillance cultures and 
sousveillant assemblage (Bakir, 2010), analogous to 
the “surveillant assemblage” of Haggerty and Ericson 
( 2000 ). 

Surveillance has strong connections to Big Data, 
where states and other large organizations, especially 
in law enforcement, collect data secretly, or at least 


WearTech™ Humanistic Intelligence Foundation, Palo Alto, CA, USA 

Corresponding author: 

Steve Mann, Phenomenal [Reality Lab™, 135 Churchill Avenue, Palo Alto, 
CA 94301, USA. 

Email: mann@eecg.toronto.edu 


Creative Commons CC-BY: This article is distributed under the terms of the Creative Commons Attribution 3.0 License (http:// 
www.creativecommons.org/licenses/by/3.0/) which permits any use, reproduction and distribution of the work without further 
permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access- 
at-sage). 








2 


Big Data & Society 


maintain some degree of exclusivity in their access to 
the data (Newell, 2013). 

Two important concepts have been proposed to help 
mitigate this one-sided nature of Big Data: (1) “giving 
Big Data a social intelligence” (Ruppert et al., 2015) 
and (2) the concept of “Personal Big Data” (Gurrin 
et al., 2014) which might more properly be called 
“little data”. Both of these concepts embody Big 
Data’s sensory counterpart that corresponds more to 
sousveillance than surveillance (Mann, 2016a). 

The veillance divide is justice denied 

A good number of recent neologisms like: "Big Data”, 
“Internet of Things” (“IoT”), “Artificial Intelligence” 
(“Al”), etc., describe technologies that aim to grant the 
gift of sight, or other sensory intelligence, to inanimate 
objects. But at the same time these inanimate objects 
are being bestowed with sight, that very same sight 
(ability to see, understand, remember, and share what 
we see) is being taken away from humans. People are 
being forbidden from having the same sensory intelli¬ 
gence bestowed upon the things around them. 

Indeed, we’re surrounded by a rapidly increasing 
number of sensors (Cardwell, 2014) feeding often 
closed and secretive “Big Data” repositories 
(Pasquale, 2015). Entire cities are built with cameras 
in every streetlight (Miao, 2015; Spielman, 2015). 
Automatic doors, handwash faucets, and flush toilets 


that once used “single-pixel” sensors now use higher- 
resolution cameras and computer vision (Iott and 
Nelson, 2005) (and with time-of-flight camera in 
US20150268 342; see also gesture-sensing shower, 
US20080256494). 

Surveillance is also widely used without regard to 
genuine privacy, i.e. with only regard to Panoptic-priv¬ 
acy. In Alberta, for example, the Privacy Commissioner 
condones the use of surveillance cameras in the men’s 
locker rooms of Calgary’s Talisman Centre where 
people are naked (David Fraser, 2007) as long as only 
the police (or other “good people”) can see the video. 
Westside recreation Centre also in Calgary, Alberta, 
also uses surveillance cameras in their men’s (but not 
women’s) locker rooms (Frakes, 2016) (Figure 1). 

While surveillance (oversight) is increasing at an 
alarming rate, we’re also seeing a prohibition on sous¬ 
veillance (undersight). 

Martha Payne, a 9-year-old student at a school in 
Scotland, was served disgusting school lunches that 
lacked nutritional value. So in 2012 she began photo¬ 
graphing the food she was served (Payne and Payne, 
2012). When she began sharing these photographs with 
others, she generated a tremendous amount of online 
discussion on the importance of good school nutrition. 
And she brought about massive improvements in the 
nutritional value of school lunches around the world. 
She also raised considerable money for charity, as a 
result of her documentary photography. But, in part 



Why No Cell Phones in 
Changing Areas? 

Would you be okay if a nude photo of you was 
posted on the internet or circulated via email? 

How about a nude photo of your spouse? 




Or, maybe your child? 



Cell phones are capable of capturing images 
and video without your knowledge. 

Westside’s “no cell phone” policy is designed to 
protect the privacy of our members and guests. 

Including yours. 


This area may be under video 
surveillance to ensure the safety of 
users by deterring acts of theft, violence 
and other criminal activity. 





Figure I. Westside Recreation Centre in Alberta, Canada, uses surveillance cameras in their men’s (but not women’s) shower/ 
locker/changerooms (Edwardson, 2016), while they also have a “no cell phone” policy allegedly to “protect the privacy” of their 
guests. 
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due to the notoriety of her photo essays, she was sud¬ 
denly banned from bringing a camera to school, and 
barred from photographing the lunches she was served 
by her school. 

While schools begin the process of installing surveil¬ 
lance cameras, students are increasingly being forbid¬ 
den from having their own cameras. And for many 
people on special diets, or with special health needs, 
using photography to monitor their dietary intake is a 
medical necessity. I proposed the use of wearable sen¬ 
sors (including wearable cameras) for automated diet¬ 
ary intake measurement in 2002 (US Pat. App. 
20020198685; Mann, 2002c). This concept is now gain¬ 
ing widespread use for self-sensing and health monitor¬ 
ing (Doherty et ah, 2013b, 2013a). 

So when people suffer from acute effects like food 
poisoning or allergic reactions, or from longer-term 
chronic effects of poor nutrition, like obesity, being for¬ 
bidden from keeping an accurate diary of what they have 
eaten is not just an affront to their free speech. It is also a 
direct attack on their health and safety. 

Neil Harbisson, a colorblind artist and musician, has 
a wearable computer vision system that allows him to 
hear colors as musical tones. And he wears his camera 
and computer constantly. Wearable computing and 
Personal Imaging (wearable cameras) are established 
fields of research (Mann, 1997), dating back to my aug¬ 
mented reality vision systems of the 1970s. I also wear a 
computer vision system and visual memory aid. 
Harbisson and Mann both have cameras attached in 
such a way as to be regarded as part of their bodies, 
and thus their passports both include the apparatus, as 
it is a part of their true selves and likenesses (see Figure 
2). And we are not alone: many people now are begin¬ 
ning to use technology to assist them in their daily lives, 
and in some ways, the transition from a surveillance 
society to a veillance society (i.e. one that includes 
both surveillance and sousveillance) is inevitable (Ali 
and Mann, 2013). 

Referring back to Martha Payne, discussed earlier in 
this article, there is often an hypocrisy of the officials 
(schools, as well as society in general) wanting to collect 
more and more data about us, while forbidding us from 
collecting data about them or about ourselves (like 
monitoring our own dietary intake, monitoring our 
exercise, or helping us see and remember what we 
see). We need to be critical of this hypocrisy because 
(1) it is a direct threat to our health, wellness, and per¬ 
sonal safety, and (2) data obtained under this hypocrisy 
lacks integrity (integrity is the opposite of hypocrisy). 
Thus it is with great joy and relief that we learn how 
Martha Payne fought back and won the right to con¬ 
tinue using photography to monitor her dietary intake, 
not only for the journalistic freedom (in the Bakir 


sense), but also for the personal safety that such self¬ 
monitoring systems can provide. 

Surveillance is a half-truth, without sousveillance 

Surveillance is the veillance of hypocrisy, in the sense 
that, as often practiced by security guards, closely 
monitoring surveillance cameras, these guards tend to 
observe and object to individuals taking pictures in the 
surveilled spaces. The opposite of hypocrisy is integrity 
(Mann et ah, 2015). 

Surveillance typically tells a story from the side of 
the security forces. When stories told from other points- 
of-view are prohibited, i.e. the capturing of evidence to 
support these other points-of-view is prohibited, the 
total captured evidence is less than the full truth 
(Mann et ah, 2015). In this sense, surveillance often 
gives rise to a half-truth. 

Justeveillance (fair sight) in Al and machine 
learning 

Much has been written about equiveillance, i.e. the 
right to record while being recorded (Manders, 2013; 
Mann et ah, 2006; Weber, 2010, 2012a), and Martha’s 
case is like so many others. 

In the context of human-human interaction, the 
transition from surveillance to veillance represents a 
“fair” (French “Juste”) sight and, more generally, fair 
and balanced sensing. 

But our society is embracing a new kind of entity, 
brought on by AI (Artificial Intelligence) and machine 
learning. Whether we consider an “AI” as a social 
entity/actor, e.g. through Actor Network Theory 
(Callon, 1999; Latour, 2005; Munro, 2009; Wood and 
Graham, 2006), or simply as a device to interact with, 
there arises the question: “Are smart things making us 
stupid?” (Morozov, 2013). 

Past technologies were transparent, e.g. electronic 
valves (“vacuum tubes”) were typically housed in trans¬ 
parent glass envelopes, into which we could look to see 
all of their internals revealed. And early devices 
included schematic diagrams and parts lists—efforts 
by the manufacturer to help end users understand 
how their products worked. 

In the present day of computer chips and closed- 
source software, manufacturers take extra effort not 
to help people understand how things work, but to 
conceal functionality: (1) for secrecy; and (2) because 
they (sometimes incorrectly) assume that their users do 
not want to be bothered by detail, i.e. that their users 
are looking for an abstraction and actually want 
“bothersome” details hidden (Burrell, 2016; Mann, 
2016b). 
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At the same time these technologies are being more 
concealing and secretive, they are also being equipped 
with sensory capacity, so that (in the Actor Network 
Theory (ANT) sense) these devices are evolving toward 
knowing more about us while revealing less about 
themselves (i.e. toward surveillance). 

Our inability to understand our technological world, 
in part through secrecy actions taken by manufacturers, 
and in part through a general apathy, leads to the use of 
modern devices through magic, witchcraft-like rituals 
rather than science (Lynn Kaarst-Brown and Robey, 
1999). This technopaganism (Stivers and Stirk, 2001) 
leads people to strange rituals rather than trying to 
understand how things work. General wisdom from 
our experts tell us to “reboot” and try again, rather 
than understand what went wrong when something 
failed (Robertson, 2011). But this very act of doing 
the same thing (e.g. rebooting) over and over again, 
expecting a different result is the very definition of 
insanity: 


“Insanity is doing the same thing, over and over again, 
but expecting different results.” (Narcotics 
Anonymous, 1981) 

In this sense, not only do modern technologies drive us 
insane, they actually require us to be insane in order to 
function properly in the technopagan world that is 
being forced upon us by manufacturers who conceal 
its workings. 

I propose 4 as a solution, a prosthetic apparatus that 
embodies the insanity for us, so that we don’t have to. I 
call this app “LUNATIC”. LUNATIC is a virtual per¬ 
sonal assistant. The user places a request to LUNATIC 
and it then “tries the same thing over and over 
again...” on behalf of the user so that the user does 
not need to himself or herself become insane. For 
example, when downloading files, LUNATIC starts 
multiple downloads of the same file, repeatedly, and 
notifies the user when the result is obtained. 
LUNATIC determines the optimum number of 



Figure 2. Harbisson and Mann with passports depicting the physical reality of their bodies as partly computational, both examples of 
people who are part technological, through the use of camera-based computer vision as a seeing aid. The Veillance Divide (e.g. when 
surveillance is the only allowable veillance) renders such people under attack as “existential contraband”—contraband by their mere 
existence. 
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simultaneous downloads. Typically this number works 
out to 2 or 3. A single download often stalls, and the 
second one often completes before the first. If too many 
downloads of the same file are initiated, the system 
slows down. So LUNATIC uses machine learning to 
detect slowed connections and makes a best guess as to 
the optimum number of times to repeat the same tasks 
over and over again. This number is called the “opti¬ 
mum insanity”, and is the level of insanity (number of 
repetitions) that leads to the most likely successful 
outcome. 

At times the optimum insanity increases without 
bound, typically when websites or servers are unreliable 
or erratic. LUNATIC is not performing a denial of 
service attack, but, rather, a “demand for service”. A 
side effect is that when large numbers of people use 
LUNATIC, erratic websites will experience massive 
download traffic, such that LUNATIC disincentivises 
insanity. 

In this sense, LUNATIC is a temporary solution to 
technopagan insanity, and ultimately will hopefully 
become unnecessary, as we transition to the age of 
Sousveillant Systems. 

Sousveillant systems 

Humanistic Intelligence is defined as intelligence that 
arises by having the human being in the feedback 
loop of a computational process (Minsky et ah, 2013). 

Sousveillant Systems are systems that are designed to 
facilitate Humanistic Intelligence by making their state 



Figure 3. Humans have senses and effectors at their informatic 
inputs and outputs. Machines have sensors and actuators at their 
informatic inputs and outputs. The signal flow paths that connect 
them are surveillance (when we’re being watched or sensed by 
the machine) and sousveillance (when we're watching or sensing 
the machine). Humanistic Intelligence (HI) requires all six of 
these signal flow paths to be present (Minsky et ah, 2013). When 
one of the six is blocked (most commonly, Sousveillance), we 
have a breakdown in the feedback loop that allows for a true 
synergy (“cyborg” state). To prevent this breakdown, Sousveillant 
Systems mandate Observability (Sousveillance). 


variables observable. In this way machines are 
“watching” us while allowing us to also “watch” 
them. See detailed description in Mann (2016a), a 
figure from which is excerpted here (Figure 3). 

Cyborg craft 

Sousveillant Systems give rise to a new form of 
Human-Computer interaction in which a machine 
can function as a true extension of the human mind 
and body. Manfred Clynes coined the term “cyborg” 
to denote such an interaction (Clynes and Kline, 1960), 
his favorite example being a person riding a bicycle 
(Clynes, 1996, personal communication). A bicycle is 
a machine that responds to us, and we respond to it 
in an immediate way. Unlike modern computers where 
feedback is delayed, a bicycle provides immediate feed¬ 
back of its state variables all which can be sensed by the 
user. 

Consider the Computer Numerical Control (CNC) 
milling machine that extends our capacity to make 
things. We begin with Computer Aided Design 
(CAD) and draw something, then send it to the 
machine, and let the machine work at it. There’s not 
much real feedback happening here between the human 
and machine. The feedback is delayed by several min¬ 
utes or several hours. 

David Pye defines “craft” (“workmanship”) as that 
which constantly puts the work at risk (Pye, 1968). As 
such, modern CNC machine work is not craft in the 
Pye sense. Nor is anything designed on a computer in 
which there is an “undo” or “edit revision history” 
functionality. 

Imagine a CNC machine that gave the kind of feed¬ 
back a bicycle does. Could we take an intimate experi¬ 
ence in craft, like a potter’s wheel, and make a CNC 
machine that works at that kind of continuous 
(“undigital”) feedback timescale? 

To answer this question, we are developing Haptic 
Augmented Reality Computer Aided Design 
(HARCAD) to create a new kind of craft called 
“Cyborg Craft” (see Figures 4 and 5). 

Here we have a sensory synergy between human and 
machine, in which the feedback loop is essentially 
instantaneous. 

Feedback delayed is feedback denied 

Let us conclude with a hypothetical anecdote set in the 
future of musical instruments, which parallels the 
author’s personal experience with scientific instruments. 
This points to a possible dystopia not so much of gov¬ 
ernment surveillance, but of machine inequiveillance. 

The year is 2067. Ken is the world’s foremost concert 
pianist, bringing his own Steinway grand piano to each 
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Figure 4. Haptics, Augmented Reality, and Computer Aided Design: A Tactile Sequential Wave Imprinting Machine (Mann, 2015, 

1992) is formed from a linear actuator connected to an antenna moved in front of a radio transmitter (microwave motion sensor). The 
actuator is fed by a signal from a lock-in amplifier connected to the moving antenna plus another stationary antenna, so the user can 
grasp, touch, hold, and feel otherwise invisible electromagnetic radio waves. In an early embodiment (Mann, 1979), radio waves are 
picked up (or reflected) by the moving metal bar of a pen plotter, and the user grasps the pen to feel the radio waves. A light bulb is 
also attached to the pen so that the user can also see the radio waves, through Persistence-of-Exposure (PoE) of the human eye, or 
photographic film. In more modern embodiments, a linear actuator drives an LED light attached to the finger. By wrestling with this 
robotic device, the user and the device together trace out waves imported into the Autodesk Fusion 360 Cloud-Based 3D CAD 
Platform for instant collaboration with others. Together with the Metavision Augmented Reality glasses, multiple people use aluminum 
foil brushes to collaboratively sculpt and shape cloud-based metaveillance waveforms to design buildings, furniture, automobiles, or 
other curvaceous products. 


concert he performs in, now that concert halls no 
longer have real pianos. Software synthesis has 
advanced to the point where none of the audience mem¬ 
bers can hear the difference. Steinway stopped making 
real pianos in 2020. Yamaha and others also switched 
to digital-only production the following year. 

Even Ken has trouble telling the difference, when 
someone else is playing, but when he plays a real 
piano himself, he can feel the difference in the vibra¬ 
tions. In essence, the performance of the Steinway 
Digital is as good as the original, but the vibrotactile 
user-interface is delayed. The tactuators installed in 
each key simulate the player’s feeling of a real piano, 
but there is a slight but noticeable delay that only the 
musician can feel. And user-interface is everything to a 
great musical performance. 

Ken no longer has access to a real piano now that his 
Steinway grand piano was water-damaged by a roof 
leak while he was away last March. He tried to buy a 
new piano but could not find one. Tucker Music had 
one in their catalog, for $6,000,000, but when Ken 


called Jim Tucker, Jim said there were no more left. 
Jim sold about 50 of them at that price, over the past 
few years, as he collects and restores the world’s last 
remaining real pianos, but no more are coming up for 
sale. 

Ken has felt that his musical performances have 
declined now that he no longer has access to a real 
piano. Software, AI, and machine learning make 
better music anyway, so there’s no longer a need for 
human musicians, anyway. 

But there has been no great advancement in music in 
recent years, now that there are no longer any great 
musicians still passionate about music for music’s 
sake. Today’s musician spends most of the time writing 
grant proposals and configuring software license ser¬ 
vers rather than playing music. 

The need for antiques to obtain truth in science 

The above story depicts a true event, except for a few 
small changes. My (not Ken’s) instrument that was 
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Figure 5. Wrestling with robots as a means for achieving Cyborg Craft: Robotic-inspired abakographic light painting (top row). 
Stephanie, age 10, wrestles with the robot while controlling the spinning of a Potterycraft wheel with her left foot. 


damaged was not a musical instrument, but, rather, a 
scientific instrument called a Lock-In Amplifier 
(Cosens, 1934; Meade, 1982; Michels and Curtis, 
1941; Stutt, 1949) made by Princeton Applied 
Research in the early 1960s. It was easy to understand 
and modify. I actually did some modifications and 
parts-swapping among several amplifiers to get some 
special capabilities for AR (augmented reality) veillance 
visualizations, such as bug-sweeping and being able to 


see sound waves, radio waves, and sense sensors and 
their capacity to sense (Mann, 2015). 

The roof leak occurred in March 2016, while the 
amplifier was running (it takes a few hours to warm 
up, and since it uses very little electricity it is best to 
leave it running continuously). 

The PAR124A is no longer available, and large 
organizations like research universities and government 
labs are hanging on to the few that remain in operation. 
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It should be a simple matter of purchasing a new 
amplifier, but none of the manufacturers are willing 
to make the modifications I require, nor are they willing 
to disclose their principles of operation to allow me to 
do so. Neither Princeton Applied Research, nor 
Stanford Research Systems (nor any other modern 
maker of lock-in amplifiers) is able to supply me with 
an instrument I can understand. 

One company claims to have equaled the perform¬ 
ance of the PAR124A, at a selling price of $17,000: 

Since the invention of the lock-in amplifier, none has 
been more revered and trusted than the PAR124A by 
Princeton Applied Research. With over 50 years of 
experience, Signal Recovery (formerly Princeton 
Applied Research) introduces the 7124. The only 
lock-in amplifier that has an all analog front end sepa¬ 
rated, via fiber, from the DSP main unit. 

Recent research findings, however, show that the 
PAR124A from the early 1960s still outperforms any 
scientific instrument currently manufactured (Wang 
et al., 2015). 

And performance alone is not the only criterion. With 
a scientific instrument we must know the truth about the 
world around us. The instrument must function as a true 
extension of our mind and body, and hide nothing from 
us. Modern instruments conceal certain aspects of their 
functionality, thus requiring a certain degree of techno¬ 
paganism (Stivers and Stirk, 2001) to operate. 

Thus we currently live in a world where we can’t do 
good science without access to antiques. 

Imagine a world in which there are no Steinway grand 
pianos anymore, a world bereft of quality, except old 
ones in need of restoration. A musician would have to 
be or hire a restorer and repair technician, and hope for 
access to one of the few working specimens that remain. 
Is this the world we want to live in? 

Science demands integrity 

Only through Sousveillant Systems and “little data” we 
can preserve the tradition of science, in the face of 
technopaganism, surveillance, and “Big-only Data”. 
A goal of our research is to produce devices that 
embody sousveillance-by-design, starting with scientific 
test instruments like lock-in amplifiers, and progressing 
toward concepts like “little data”. To that end, let us 
hope that we can build sousveillance into our net¬ 
worked world, starting with instruments. 

Declaration of conflicting interests 

The author(s) declared no potential conflicts of interest with 
respect to the research, authorship, and/or publication of this 
article. 


Funding 

The author(s) received no financial support for the research, 

authorship, and/or publication of this article. 

Notes 

1. Lyon (2007); Norris et al. (2002). 

2. Ali et al. (2013b, 2013a); Bakir (2010, 2009); Cardullo (2014); 
Fernback (2013); Fletcher et al. (2011); Ganascia (2010); 
Manders (2013); Mann (2002b); Mann et al. (2003); 
Mortensen (2014); Quessada (2010); Reilly (2015); 
Reynolds (2011); Vertegaal and Shell (2008); Weber (2012b). 

3. Mann (2016a); Mann and Ferenbok (2013). 

4. This began as an interventionist/awareness-raising effort, 
but could evolve into a useful field of research. 
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